The volume of data generated in modern systems has grown exponentially in recent years, and log files account for a significant portion of this growth. Components that generate logs include:

Companies collect and store logs to help troubleshoot incidents, find the root cause of performance problems, investigate security breaches, or adhere to compliance requirements.

However, logs can consume considerable disk space. The size of individual log files can spiral out of control, and available disk space can be taxed. Without proper controls, system performance issues will arise.

This is the impetus behind log rotation: a controlled way of removing or archiving older logs to make room for new ones.

This article will cover the basics of log rotation—why it’s important, and what you can do with your older log files. We’ll also look briefly at Falcon LogScale: a modern, cloud-based log management system.

Log rotation is the process of controlling the size of log files. When an existing log file reaches a certain threshold—usually a maximum file size, age, or number of records—the associated application will rename it, create a new file with the original name, and continue writing events to the new file. The application may compress the older log file, archive it, or even delete it. Log rotation is a quick, automated process that takes seconds to complete, and some applications run a dedicated thread to handle it.

A common naming style with log rotation is to append the timestamp of the rotation to the older file name. For example, if the log file name is mylogfile.log, the rotated log file will be named mylogfile_xxxxxxxx.log (where xxxxxxxx is a date or timestamp), and the newly created file will be named mylogfile.log. The resulting log file folder may show entries like this:

Sometimes, you may also see suffixes appended to older log files instead of timestamps. For example, the SQL Server error log file name is ERRORLOG, and the older files will be named ERRORLOG.1, ERRORLOG.2, ERRORLOG.3, and so on; the higher the number, the older the file.

You can usually control log rotation settings by configuring the application or the operating system with certain criteria. Once the settings are saved, a log rotation will trigger when these criteria are met. You can also opt to turn log rotation off in certain cases, but this should only be a temporary measure. Some applications allow you to set log file names, specify where they should be saved, and even disable logging entirely.

Sometimes it’s necessary to keep older logs locally, as application events can span over multiple rotated files. You can compress older files to save space on the server or configure them for removal after a specified period. You can move them to archival storage devices, backup servers, cloud-based object storage media like S3, or a log management system for longer-term retention.

As mentioned before, log files can be useful for troubleshooting errors, investigating security incidents, and assessing your IT environment. But without log rotation in place, noisy logs capturing network traffic or authentication events can quickly become a problem.

If you don’t configure your log files for compression or deletion, they will grow until the server has run out of disk space. Critical services will stop functioning, and it can cause gaps in the logs, as the application can only write new events when space is available. If an incident occurs during this time, you may not have any record of it.

Even if disk space isn’t full, your server can still run into performance issues. If it doesn’t have enough memory, then opening, reading, and writing to large files (sometimes GBs in size) will take considerably longer. Manual operations like tail, head, or grep can be time consuming, and this isn’t ideal when you are investigating a critical issue.

Log management solutions can often struggle to process large log files. Logging agents will take longer to stream the data over the network. The log management software will take longer to parse and index the data. All of this can delay real-time alerting. For example, we can imagine a scenario where authentication logs contain evidence of a password spray attack. Due to the large file size, however, it takes thirty minutes for the security information and event management (SIEM) solution to find the attack and fire an alert. Obviously, this would not be an ideal situation.

Unless you can provide unlimited storage for your servers, then you must have a strategy for handling older log files. Some businesses take different approaches for different types of log files depending on their use cases, importance, and legal obligations.

The easiest thing to do with rotated log files is to delete them. It’s easy to configure, clears space on the server, and cuts storage costs. However, since log files often contain important events—and you don’t always know when you’ll need them—this can be risky. Also, you may lose important information when an event spans multiple log files. Depending on the industry of your business and compliance needs, deleting old log files may not be an option.

You can compress log files as part of the rotation process and maintain older files on the server. This is common in Linux systems, and you’ll typically see compressed log files with .gz extensions.

Archiving saves older logs to a separate storage system for long-term retention. This could be a purpose-built backup server with external storage (such as SAN), cheaper cloud storage (such as AWS S3), or offsite storage on tape.

Most IT operations depend on a central system to store all their logs. These systems allow users to search, analyze, correlate, and visualize events from log files. This is far simpler than manually processing log files from the file system. Log management systems can show historical trends, compare system states, and provide real-time alerts for critical events.

Remember that log rotation is not a substitute for a log management system. The two complement each other. You can rotate, archive, or remove log files once consumed and stored by the log management system.

Regardless of where you send older log files, you need to secure them. This ensures any sensitive information contained in the log files is protected and the files can’t be tampered with.

Falcon LogScale Community Edition (previously Humio) offers a free modern log management platform for the cloud. Leverage streaming data ingestion to achieve instant visibility across distributed systems and prevent and resolve incidents.

Falcon LogScale Community Edition, available instantly at no cost, includes the following: